OK, let's start from the beginning: I just read that in 2018 the US government announced a new security clearance program - including for individuals in civilian roles - which would run "continuous evaluations" of all applicants, thanks to machine learning technology. The article itself highlights the obvious risks of such a system "going off the rails", but the really interesting questions here are: At least in certain cases, we may never know the answer to the first question because, as the article says, certain systems "offer little to no insight as to how their highly accurate predictions are actually made". But hey, we are only talking of national security, no big deal right? So let's focus on the second question. The article does correctly acknowledge my first thought when I read its title: "if the system works, it might actually generate deeper problems still".